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ABSTRACT 

This  paper  describes  ongoing  research  by  Georgia  Tech  into  the  challenges  of  tasking  and  controlling 
heterogonous  teams  of  unmanned  vehicles  in  mixed  indoor/outdoor  reconnaissance  scenarios.  We  outline  the  tools  and 
techniques  necessary  for  an  operator  to  specify,  execute,  and  monitor  such  missions.  The  mission  specification 
framework  used  for  the  purposes  of  intelligence  gathering  during  mission  execution  are  first  demonstrated  in  simulations 
involving  a  team  of  a  single  autonomous  rotorcraft  and  three  ground-based  robotic  platforms.  Preliminary  results 
including  robotic  hardware  in  the  loop  are  also  provided. 
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1.  INTRODUCTION 


One  of  the  most  promising  applications  for  microautonomous  robotic  systems  (e.g.,  Figure  1)  lies  in  the  domain  of 
distributed  reconnaissance.  These  systems’  potentially  decentralized  processing,  innocuous  size,  distributed  sensing 
capabilities,  and  low  cost  will  afford  tomorrow’s  soldier  a  powerful  tool  for  situational  awareness.  This  paper  describes 
one  such  indoor/outdoor  reconnaissance  scenario  designed  and  implemented  as  part  of  the  Army  Research  Lab’s  (ARL) 
Micro  Autonomous  Systems  and  Technology  (MAST)  initiative  [1]. 

While  the  scenario  described  is  geared  towards  microautonomous  systems,  initial  research  has  focused  on  larger 
surrogate  platforms  until  the  first  microautonomous  vehicles  become  available.  In  the  scenarios  that  serve  as  the  focus 
of  this  paper,  an  unmanned  aerial  vehicle  is  used  first  to  scout  the  exterior  of  a  target  building,  discover  an  entrance 
point,  and  then  utilize  that  ingress  to  locate  a  target  of  interest.  Once  the  target  has  been  identified,  the  aerial  vehicle 
then  guides  the  team  of  ground  vehicles  into  the  building  and  into  the  proximity  of  the  target  of  interest  using  a 
controlled  formation.  Finally,  when  contact  has  been  made,  the  ground  vehicles  form  a  mobile,  distributed  sensor 
network  suitable  for  intelligence  gathering,  including  visual  Simultaneous  Localization  and  Mapping  (SLAM)  for 
reconnaissance. 

The  remainder  of  the  paper  is  structured  as  follows.  First,  a  review  of  microautonomous  robotic  systems  is 
presented.  Then,  an  overview  of  the  MissionLab  robot  mission  specification  system  is  provided  and  the  manner  by 
which  an  operator  may  specify  the  particular  reconnaissance  scenario  that  serves  as  the  focus  of  this  paper  within 
MissionLab  follows.  A  simulation-based  verification  of  this  exemplar  mission  is  covered  in  section  4  while  the 
description  of  the  scenario  running  upon  hardware  platforms  appears  in  Section  5.  Finally,  we  conclude  by  reviewing 
the  major  contributions  of  this  paper. 


2.  RELATED  WORK 


A  significant  body  of  work  has  begun  to  accrue  which  explores  the  viability  of  microautonomous  vehicles  in  various 
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Figure  1.  Artist's  rendition  of  a  microautonomous  robot  inspired  by  a  spider  (Photo  from  BAE  Systems). 

contexts.  Much  of  this  existing  research  examines  microautonomous  vehicles  from  a  feasibility  standpoint.  That  is 
to  say,  these  avenues  of  research  focus  on  the  mechanical,  electrical,  and  control  requirements  necessary  for 
microautonomous  vehicles.  Examples  include  research  into  the  physical  characteristics  necessary  for  microautonomous 
underwater  vehicles  [2],  the  power  requirements  of  these  vehicles  [3],  and  the  various  approaches  towards  sensing  [4]  [5] 
and  control  [6]  of  microautonomous  robotic  systems. 


As  the  technical  challenges  of  deploying  microautonomous  vehicles  in  the  land,  sea,  and  air  are  identified  and  overcome, 
an  increasing  number  of  researchers  have  begun  examining  the  space  of  potential  applications  and  missions  to  which 
these  vehicles  may  be  suited.  This  paper  serves  as  one  such  example.  Others  include  Tester,  et  al.’s  research  using 
teams  of  microautonomous  underwater  vehicles  to  monitor  hydrographic  features  within  shallow  waters  [7].  Davis  et 
al.,  on  the  other  hand,  provide  an  early  example  of  possible  surveillance  applications  for  microautonomous  unmanned 
aerial  vehicles  [8].  While  surveillance  scenarios  such  as  this  one  are  a  common  mission  type  for  microautonomous  and 
traditional  robotic  systems,  less  frequently  are  heterogeneous  robot  teams  involved  in  these  scenarios.  When 
heterogeneous  teams  of  unmanned  systems  are  deployed,  one  of  the  most  common  configurations  is  a  combination  of 
unmanned  aerial  vehicles  and  unmanned  ground  vehicles  (e.g.  [9]).  In  these  scenarios,  the  unmanned  aerial  vehicles 
take  advantage  of  the  mobility  and  elevation  to  provide  intelligence  to  the  deployed  ground  vehicles  and  guide  them  to 
areas  of  interest.  The  scenario  described  in  this  paper  takes  a  similar  approach  in  which  the  unmanned  vehicle  identifies 
and  tracks  the  target  in  service  of  a  group  of  unmanned  vehicles  within  the  mission  area. 


3.  SCENARIO 

The  remainder  of  this  paper  describes  one  potential  indoor/outdoor  reconnaissance  mission  in  detail  including  the 
behavior  of  the  individual  robots  that  participate  in  the  mission,  the  behavior  of  the  target  of  interest  in  this  scenario,  and 
details  concerning  the  sensory  processing  that  will  be  performed  by  the  vehicles  in  the  mission.  Finally  both  simulation 
results  as  well  as  hardware  runs  of  the  scenario  are  described. 

As  mentioned  earlier,  in  the  scenario  described  in  the  paper,  an  unmanned  aerial  vehicle  is  used  to  survey  the 
exterior  of  an  unknown  building,  discover  a  point  of  ingress,  and  then  utilize  that  ingress  to  locate  a  target  of  interest. 
Potential  ingress  points  for  the  ground  vehicles  are  also  located  by  the  aerial  vehicle.  If  and  when  the  target  has  been 


Figure  2.  Environment  for  the  experiment,  with  the  three  ground  robots  in  place  and  the  aerial  vehicle  perched. 


identified,  the  aerial  vehicle  then  informs  the  ground  vehicles  in  proximity  of  the  building  of  the  target’s  location, 
subset  of  the  ground  vehicles  then  proceed  to  advance  towards  that  position  until  contact  with  the  target  is  made. 


Figure  3.  Indoor  environment  that  the  scenario  takes  place  in,  including  the  partition  separating  the  room  into  two  parts. 


The  other  ground  vehicles  also  enter  the  building  but  explore  the  environment.  Once  visual  contact  has  been  made  of  the 
target,  the  ground  vehicles  spread  out  across  the  environment  so  as  to  generate  a  map  of  the  building,  utilizing 
distributed  SLAM  techniques  being  developed  by  our  collaborators  for  similar  computationally  limited  platforms  [10]. 


The  robots  used  in  this  scenario  consist  of  a  team  of  three  platforms,  each  equipped  with  a  color  camera,  Zigbee- 
based  wireless  communication,  and  a  single  forward-facing  IR  sensor.  As  such,  these  surrogate  platforms  are  sensor- 
limited  as  the  real  microautomous  robots  will  also  be.  In  particular,  the  odometry  available  from  the  ground  robots  is  as 
limited  as  that  of  a  small  crawler  might  be.  In  addition,  a  single  autonomous  rotorcraft  equipped  with  a  color  camera  and 
wireless  communication  serves  as  the  target  designator  and  tracker  for  the  team.  The  scenario  takes  place  inside  and  just 
adjacent  to  a  building  approximately  9x6  meters  large.  A  small  open  window  accessible  from  the  outside  of  the  building 
serves  as  the  entrance  for  the  UAV  so  that  it  may  observe  the  interior.  In  addition  to  this  window,  a  freight-loading  door 
leading  into  the  building  is  partially  open  allowing  the  unmanned  ground  vehicles  to  enter  should  the  UAV  identify 
anything  of  interest  in  the  building.  Note  that  in  this  preliminary  implementation,  the  ground  robots  begin  just  inside  the 
building  and  the  aerial  vehicle  is  perched  just  below  a  vent  (Figure  2).  Finally,  the  interior  of  the  building  is  partially 
divided  by  a  partition  splitting  the  room  into  two  sections  (Figure  3). 

4.  SCENARIO  IMPLEMENTATION 


We  now  describe  a  preliminary  implementation  of  the  above  scenario  that  is  tested  in  simulation  as  well  as  on  real 
robots.  Note  that  some  of  the  elements,  such  as  ingress  detection  by  the  aerial  vehicle  remain  to  be  implemented  as  the 
project  progresses.  In  order  to  generate  missions  for  each  individual  member  of  the  team,  MissionLab ,  a  software  toolset 
that  allows  a  user  to  compose  multi-robot  mission  through  a  graphical  interface  was  used  [11].  MissionLab  allows  a 
commander  to  generate  finite  state  acceptor  (FSA)  based  mission  plans  that  can  be  compiled  and  executed  on  each 
individual  robot.  In  MissionLab' s  FSA-based  mission  representation,  actions  (behaviors)  are  represented  as  circles, 
while  perceptual  triggers  (conditions  for  executing  the  next  action)  are  denoted  as  arrows  between  behaviors.  A  mission 
consists  of  a  sequence  of  these  behaviors  and  the  perceptual  triggers  that  result  in  transitions  between  them.  In  this 
particular  scenario,  each  of  the  robots  is  tasked  with  a  slightly  different  mission.  The  remainder  of  the  section  describes 
the  composition  of  the  individual  FSAs  used  in  the  scenario. 

Figure  4  depicts  the  FSA  used  in  the  scenario  to  describe  the  unmanned  rotorcraft’ s  mission.  The  first  stage 
instructs  the  vehicle  to  approach  the  window  of  the  building  and  then  once  it  arrives  there  to  alert  the  operator  that  it  will 
begin  to  search  for  the  target.  At  the  window,  the  robot  begins  searching  for  the  target  by  panning  its  camera  (by 
rotating  about  its  yaw  axis).  If  the  target  is  found,  the  robot  reports  acquisition  to  both  the  operator  and  the  ground 
vehicles.  Further,  the  position  of  the  target  is  sent  to  the  UGVs  on  the  team  so  that  they  may  be  able  to  find  the  target 
upon  entering  the  building.  Finally,  the  mission  specifies  that  if  the  rotorcraft  ever  loses  track  of  the  target,  is  should 
resume  its  search  until  it  is  found  again. 

The  general  FSA  specifying  the  mission  for  the  unmanned  ground  vehicles  in  the  scenario  can  be  seen  in  Figure  5. 
While  the  mission  specification  for  each  UGV  differs  slightly,  all  three  follow  the  same  general  plan,  which  consists  of 
three  independent  phases.  In  the  first  phase,  the  unmanned  ground  vehicle  nears  the  entry  point  of  the  target  building 
awaiting  the  signal  from  the  rotorcraft  indicating  that  the  target  of  interest  has  been  identified  within.  Once  the  location 
of  the  target  of  interest  has  been  transmitted  to  the  UGVs,  the  second  phase  of  the  mission  begins.  The  vehicles  are  then 
instructed  to  enter  the  building  in  a  loose  formation  and  begin  traveling  to  the  target  while  avoiding  any  obstacles  in  their 
path.  This  loose  formation  is  the  result  of  a  repulsive  sphere  around  each  robot  to  ensure  that  they  remain  a  certain 
distance  away  from  one  another  coupled  with  an  attractive  sphere  to  make  sure  that  if  the  robots  do  not  stray  too  far  from 
their  teammates.  This  behavior  creates  a  formation  that  is  not  dependent  on  the  number  of  robots,  and  hence  additional 
robots  can  be  added  as  needed  without  modifying  the  mission. 

The  unmanned  rotorcraft  reports  updates  on  the  position  of  the  target  of  interest  to  the  ground  vehicles  to  assist  in 
maintaining  the  target’s  location  in  the  event  that  the  target  is  mobile.  Once  the  robots  have  encountered  the  target,  a 
report  is  sent  to  the  commander  and  the  third  phase  of  the  mission  begins.  This  third  phase  consists  of  the  UGVs 
spatially  distributing  themselves  around  the  building  and  at  varying  intervals  stopping  and  visually  scanning  the  building 
in  order  to  generate  a  map  that  may  be  of  use  to  accompanying  human  ground  forces.  Note  that,  as  mentioned  in  the 
previous  section,  a  subset  of  the  robots  do  not  enter  the  second  stage  (movement  in  a  formation  towards  the  target); 
instead,  they  enter  the  building  and  begin  the  third  stage  directly  and  wander  around  the  environment  in  order  to  aid 
map-building.  Note  that  all  movement  behaviors  contain  sub-behaviors  responsible  for  obstacle  avoidance. 
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Figure  4.  Mission  FSA  used  for  the  rotorcraft  in  the  demonstration  scenario.  Initially  the  UAV  travels  to  the  opening  in  the 
building  and  then  proceeds  to  identify  and  track  the  target  of  interest  visually. 
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Figure  5.  Typical  mission  for  unmanned  ground  vehicles  used  in  the  demonstration  scenario.  The  vehicles  are  first  ordered  to 
travel  to  the  opening  of  the  building.  Upon  receiving  the  position  of  the  target  of  interest  from  the  UAV,  a  subset  of  the 
ground  vehicles  travel  in  formation  until  they  encounter  the  target.  Once  they  have  encountered  the  target,  the  team  joins  the 
other  ground  robots  and  explores  the  environment  to  conduct  a  visual  survey  in  support  of  mapping. 


5.  SCENARIO  SIMULATION 


In  order  to  evaluate  the  scenario  outlined  above,  a  simulation-based  verification  was  conducted.  This  verification  took 
place  via  the  integration  of  two  software  packages.  MissionLab  was  used  to  specify  the  mission  as  shown  above  and  to 
run  the  robot  executables  (a  compiled  software  program  which  realizes  the  mission  specified  in  MissionLab  for  each 
robot.)  The  Gazebo  simulation  platform  [12]  was  used  to  simulate  the  motor  and  sensory  capabilities  of  the  platforms  in 
a  3D  physics-based  simulation  environment.  The  remainder  of  this  section  will  discuss  a  typical  execution  of  the  target 
scenario.  A  movie  of  the  scenario  described  here  can  be  found  at  [13]. 

Figures  6  to  8  depict  a  typical  mission  run  in  the  Gazebo  simulation  environment.  Figure  6  depicts  the  initial 
configuration  of  the  robots  in  the  scenario.  The  team  of  three  UGVs  initially  waits  at  a  partially  open  garage  leading  to 
the  building  interior.  The  UAV  is  positioned  north  of  a  potential  ingress  to  the  building.  As  the  mission  begins,  the 
UAV  proceeds  to  the  window  in  order  to  observe  any  potential  targets  inside.  Once  the  UAV  has  detected  the  target,  it 
reports  its  position  to  the  UGV  team  which  then  proceeds  to  enter  the  building  and  head  towards  the  target  along  each  of 
the  two  potential  routes  around  the  partition  (Figure  7).  The  scenario  enters  its  final  phase  when  one  of  the  UGVs  makes 
contact  with  the  target  and  reports  that  contact  has  been  made.  Upon  receiving  this  report,  the  UGV  team  begins  to 
distribute  themselves  around  the  environment  in  order  to  both  track  future  movement  of  the  target  and  to  perform  SLAM 
using  their  onboard  cameras. 


Figure  6.  Initial  scenario  configuration  within  the  MissionLab/Gazebo  simulation  environment.  The  team  of  UGVs  stand 
ready  at  the  entrance  of  the  building  while  the  UAV  begins  moving  towards  the  window  for  an  interior  view. 


Figure  7.  The  UAV  has  positioned  itself  at  the  window  and  has  identified  the  target.  The  team  of  UGVs  enter  the  building  in  a 
loose  formation.  One  of  the  ground  vehicles  begins  to  move  to  the  target,  while  the  rest  spread  out  and  explore  the 
environment.  As  the  target  moves,  the  UAV  provides  updates  to  the  UGVs  within  the  building.  (The  UAV  is  partially 
obscured  by  the  wall  due  to  the  camera  angle  of  the  simulation.) 


Figure  8.  After  encountering  the  target,  all  of  the  robots  explore  the  environment  in  order  to  ensure  the  target  may  be  tracked 
as  well  as  to  generate  additional  intelligence  in  the  form  of  a  map. 


Figure  9.  Left:  The  Skybotix  CoaX  autonomous  helicopter  used  in  the  hardware  based  verification  of  the  scenario  (Photo  from 
Skybotix).  Right:  Wowwee's  Rovio  telepresence  platform  (Photo  from  Wowwee). 


Figure  10.  A  Cricket  indoor  localization  developed  by  MIT  and  manufactured  by  Crossbow.  Each  listener  mote  determines  is 
position  based  upon  the  position  of  several  beacon  motes. 


6.  HARDWARE  VERIFICATION 


Once  the  simulation-based  verification  of  the  scenario  had  been  conducted,  we  transitioned  the  mission  onto  hardware 
platforms.  As  discussed  earlier,  this  scenario  was  developed  in  the  context  of  the  goals  of  the  Micro  Autonomous 
Systems  and  Technology  initiative.  As  the  first  microautonomous  platforms  being  developed  were  not  available  at  this 
time,  we  used  larger  surrogate  platforms  as  the  means  of  testing.  The  unmanned  rotorcraft  that  serves  in  the  target 
identification  role  was  realized  via  the  Skybotix  Technologies  CoaX  platform  (Figure  9).  The  CoaX  platform  is  capable 
of  fully  autonomous  or  teleoperated  flight  and  comes  equipped  with  a  color  camera,  inertial,  and  sonar  sensors. 

The  platforms  that  serve  as  the  surrogate  ground-based  microautonomous  vehicles  were  Wowwee’s  telepresence 
robot  Rovio  (Figure  9).  While  the  Rovio  platform  is  equipped  with  a  maneuverable  color  camera,  the  difficulty  in 
detecting  and  navigating  around  obstacles  using  this  non-stereo  camera  necessitated  a  precomputed  map  of  all  obstacles 
and  walls  that  is  used  only  during  this  developmental  stage  (i.e.,  prior  to  integration  of  SLAM  capability).  This  map  was 
provided  to  each  ground  robot  so  as  to  compute  ‘virtual’  obstacle  locations.  These  virtual  obstacles  were  then  treated  as 
sensed  obstacles  as  appropriate  given  the  robot’s  currently  estimated  position.  In  addition,  the  single  forward-facing  IR 
sensor  was  used  for  obstacle  avoidance  when  the  robots  became  too  close  to  an  obstacle. 

To  assist  in  pose  estimation  and  to  provide  a  means  of  computing  ground  truth  localization  for  evaluation  of  visual 
SLAM  mapping  performance,  each  Rovio  was  also  equipped  with  the  Cricket  indoor  localization  sensor,  manufactured 
by  Crossbow  (Figure  10).  Listener  Cricket  motes  compute  their  current  position  based  upon  active  radio  and  sonar  pings 
generated  by  groups  of  beacon  motes  mounted  around  the  building.  These  listeners  estimate  their  position  based  upon 
the  known  distance  of  three  or  more  of  these  beacons.  In  addition,  when  only  two  beacons  were  available,  prior  altitude 


information  was  used  to  constrain  the  solution  and  obtain  new  pose  information.  This  position  estimate  is  then  fused 
with  the  Rovio’s  odometry  using  an  extended  Kalman  filter  and  logged  to  provide  ground  truth  positioning  for  map 
quality  estimation. 

Note  that  the  Cricket-based  localization  system  provides  three-dimensional  position  information  but  does  not 
provide  yaw,  pitch,  or  roll.  In  order  to  estimate  yaw,  consecutive  Cricket  positions  were  averaged  over  two  time 
windows  and  the  angle  between  the  average  positions  was  calculated.  If  the  resulting  two  positions  were  far  enough 
apart  (according  to  an  empirically-derived  threshold),  then  the  yaw  information  was  added  to  the  Cricket  readings  and 
incorporated  into  the  Kalman  filter.  The  target  of  interest  in  this  scenario  was  realized  via  a  separate  human-controlled 
Rovio  platform  with  a  brightly-colored  object  place  on  it.  Finally,  target  identification  on  both  the  Rovio  and  CoaX 
platforms  was  conducted  via  color  blob  tracking  implemented  using  OpenCV,  an  open-source  computer  vision  library 
[14]. 

Figures  11  to  13  show  one  example  of  the  hardware-based  runs  of  the  reconnaissance  scenario.  A  video  of  the 
hardware-based  run  can  be  found  at  [15].  Each  of  the  figures  shows  the  scenario  at  analogous  points  in  time  to  the 
simulation  shown  in  Figures  6  to  8.  In  the  hardware-based  runs  of  the  scenario,  both  the  team  of  Rovios  and  the  CoaX 
begin  outside  of  the  building  (Figure  11).  Similar  to  the  simulation  of  the  scenario,  the  mission  begins  with  the  CoaX 
successfully  identifying  the  target  via  the  building’s  window  and  provides  a  constant  stream  of  target  status  updates  as 
the  target  moves  about  the  building  (Figure  12).  The  Rovio  platforms  use  these  position  updates  to  navigate  into  the 
building,  and  a  subset  of  the  robots  travel  to  the  target.  As  in  the  simulation,  once  the  target  has  been  reached,  all  of  the 
UGVs  explore  the  environment  to  collect  camera  imagery  suitable  for  map  building  (Figure  13). 


Figure  11.  Initial  scenario  configuration  of  the  real  robot  run.  The  team  of  UGVs  stand  ready  just  inside  the  building  while 
the  UAV  (off-camera)  attempts  to  track  the  target. 


Figure  12.  The  UAV  has  identified  the  target,  and  the  team  of  UGVs  begins  their  mission.  One  of  the  ground  vehicles  begins 
to  move  to  the  target,  while  the  rest  explore  the  environment.  As  the  target  moves,  the  UAV  provides  updates  to  the  UGVs 
within  the  building.  (The  UAV  is  outside  of  the  frame  toward  the  upper  left.) 


Figure  13.  The  robot  arrives  at  the  target.  After  encountering  the  target,  all  of  the  robots  further  explore  the  environment  in 
order  to  ensure  the  target  may  be  continuously  tracked  in  addition  to  generating  additional  intelligence  in  the  form  of  a  map. 


7.  CONCLUSIONS 


The  use  of  microautonomous  vehicles  in  surveillance  scenarios  is  one  of  the  most  promising  application  domains  for 
these  platforms.  Significant  research  must  be  done  in  order  to  measure  the  effectiveness  of  those  platforms  in  such 
coordinated  teaming  scenarios,  however.  We  have  presented  one  such  indoor/outdoor  surveillance  scenario  utilizing 
both  autonomous  unmanned  ground  and  unmanned  aerial  vehicles.  We  described  how  such  a  mission  can  be  specified 
in  the  MissionLab  multi-robot  specification  environment.  This  scenario  was  first  demonstrated  in  simulation  to  verify 
operator  intent.  Once  the  target  scenario  was  operating  appropriately  in  simulation,  a  hardware-based  verification  was 
successfully  demonstrated.  This  particular  mission  is  only  an  example  of  the  large  potential  space  of  missions  that  can 
be  created  and  tested  using  the  software  tools  described  herein  (more  or  less  robots,  different  building  layouts,  etc.) 

Future  work  will  involve  the  complete  implementation  of  the  scenario,  including  ingress  detection,  utilization  of 
additional  ground  robots  that  will  flock  towards  the  target,  and  integration  with  distributed  visual  SLAM.  Beyond  the 
current  scenario,  future  work  will  also  look  at  additional  domains  in  which  the  promise  of  microautonomous  vehicles 
will  be  most  beneficial. 
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